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In this paper we introduce and exploit the real replica approach for a minimal generalization of 
the Hopfield model, by assuming the learned patterns to be distributed accordingly to a standard 
unit Gaussian. We consider the high storage case, when the number of patterns is linearly diverging 
with the number of neurons. We study the infinite volume behavior of the normalized momenta of 
the partition function. We find a region in the parameter space where the free energy density in the 
infinite volume limit is self- averaging around its annealed approximation, as well as the entropy and 
the internal energy density. Moreover, we evaluate the corrections to their extensive counterparts 
with respect to their annealed expressions. The fluctuations of properly introduced overlaps, which 
act as order parameters, are also discussed. 

PACS numbers: 87.85.dq, 89.75.-k, 75.50.Lk 

I. INTRODUCTION 

In the last twenty years, from the early work by Hopfield (2§| and the, nowadays historical, theory of Amit, 
Gutfreund and Sompolinsky (AGS) [1, 0)0] to the modern theory for learning @, EH], the neural networks, thought of 



as spin glasses with a Hebb-like "synaptic matrix" 25], became more and more important in several different contexts, 
as artificial intelligence 26], cognitive psychology 27], problem solving [1, H3, and so on. Despite their fundamental 
role, and due to their very difficult mathematical control, very little is rig orously known about these networks. Along 
the years several contributions appeared (e.g. fH. Isj. ITo . [Til H3 , HH, [Hf HH Ha]), often motivated by an increasing 
understanding of spin-glasses (e.g. 0, [20, l3ll. |36|) and the analysis at low level of stored memories has been 
achieved. 

However in the high level of stored memories, fundamental enquiries are still rather incomplete. Furthermore, general 
problems as the existence of a well defined thermodynamic limit are unsolved, in contrast with the spin glass case 

tup. 

In this paper, we introduce some techniques (essentially in the real replica framework) developed for the spin glass 
theory (see e.g. @, 0, [H, HH); in order to give a description of the Hopfield model in the high temperature region 
with high level of stored memories (i.e. patterns), by no use of replica trick [30(. We take the freedom of allowing the 
learned patterns to take all real values, their probability distribution being a standard Gaussian Af[0, 1], and we refer 
to this minimal generalization as analogical Hopfield model, to stress that the memories are no longer discrete as in 
standard literature. 

Within this scenario, we exploit the moment method in order to prove bounds on the critical line for the ergodic 
phase and give the explicit expression for all the thermodynamical quantities in the infinite volume limit, in complete 
agreement with AGS theory. Furthermore we show in a simple way self-averaging of free and internal energy and 
entropy per site and calculate their extensive fluctuations around the annealed expressions, in analogy with what was 
found by Aizenman, Lebowitz and Ruelle [3^ ] for the Sherrington-Kirkpatrick model in the ergodic region. 
We investigate also about the overlap fluctuations, in the ergodic region. The paper is organized as follows. In section 
HI! we define the analogical Hopfield model, and show that it is equivalent to a bipartite spin glass, where one party 
is described by Ising spins and the other by Gaussian spins. We introduce also the main thermodynamic quantities 
and their annealed approximation. In the next section 1X111 we introduce overlaps for replicas of the Ising spins and the 
Gaussian spins, and show how they enter in the expression of thermodynamic quantities, as for example the internal 
energy. In section IV we state our main results about the validity of the annealed approximation, and establish the 
fluctuations of the extensive thermodynamic variables and the overlaps. In section V we study the momenta of the 
normalized partition function in the infinite volume limit, and prove our results about the annealed approximation. 
In section VI we prove the log-normality of the limiting distribution for the partition function, and prove the rest of 
our results. Finally, section VII is devoted to some conclusion and outlook for future developments. 
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II. DEFINITION OF THE MODEL 



We introduce a large network of N two-state neurons (1, .., N) 5 i — ► <7i = ±1, which are thought of as quiescent 
(sleeping) when their value is —1 or spiking (emitting a current signal to other neurons) when their value is +1. They 
interact throughout a synaptic matrix Jy- defined according to the Hebb rule for learning 

■i, EW- (i) 

Each random variable £ M = {£,1 , ■ ■ , represents a learned pattern and tries to bring the overall current in the 
network (or in some part) stable with respect to itself (when this happens, we say we have a retrieval state, see e.g. 
Q). The analysis of the network assumes that the system has already stored p patterns (no learning is investigated 
here), and we will be interested in the case in which this number increases linearly with respect to the system size 
(high storage level), so that p/N — > a as N — ► oo, where a > is a parameter of the theory denoting the storage 
level. 

In standard literature these patterns are usually chosen at random independently with values ±1 taken with equal 
probability 1/2. Here, we chose them as taking real values with a unit Gaussian probability distribution, i.e. 

M#) = -^e-«n 2 / 2 . (2) 



Of course, avoiding pathological case, in the high storage level and in the high temperature region, the results should 
show robustness with respect to the particular choice of the probability distribution and we should recover the standard 
AGS theory. The use of a Gaussian distribution has some technical advantages, as it allows to easily borrow powerful 
techniques from the spin glass case. The physical interpretation is very simple. While the patterns with ±1 values 
describe "images" with white or black pixels, respectively, the patterns with continuous values describe "gray" pixels 
with a continuously variable luminosity from — oo (completely black) to +oo (completely white). 
The Hamiltonian of the model involves interactions between any couple of sites according to the definition 



p N 
H=l i<j 

By splitting the summations = \ Ylij ~ \ Yli <%> we can wr he the partition function in the following form 

, a p N a p N 

Z N ,M = E «P (4 £ £ - ^ E E(^) 2 ) = Zn,M Oe- ^ (4) 

u /i— 1 ij i 

where f3 > is the inverse temperature, and denotes here the level of noise in the network. We have defined 

a P N 
a fj,= l ij 

Notice that the last term at the r.h.s. of eq. ^ does not depend on the particular state of the network. 

As a consequence, the control of the last term can be easily obtained. In fact, let us define the random variable 

so that 

p N 

/- = ^EE(^) 2 - (6) 



N 



Then we have 



\nZ N . p {P;0 = ln^v, P (/3;0 - f jW- (7) 

Since E/jv = P we have immediately limjv— >0 o(1/^V)E/jv = ot. On the other hand, is a sum of independent random 
variables and therefore, by the strong law of large numbers, we have also lim7v_ +00 (l/./V)//v = a, ^-almost surely. 
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Furthermore, by the central limit theorem, we have, in distribution, lirn/v— »oo(/jv — E/jv) = v2a X, X being a standard 
Gaussian 7V(0, 1). In fact, 

p N p N 

E (/-)-^EEEE E (^) 2 (^) 2 ) =p 2 + 2 ^ ( 8 > 

li i v j 



so that E(/^,) — E(/tv) 2 = 2p/N, which in the thermodynamic limit gives the result. 

Consequently we focus just on Z(/3;£). Let us apply the Hubbard-Stratonovich lemma [l|| to linearize with respect 
to the bilinear quenched memories carried by the If we define the "Mattis magnetization" Q m M as 



1 W 



p AT 
we can write 

Z NjP (f3;Z) = 2^exp(— 2^«V = 2^ / (11 ~/f= )exp(V/3AT ^ 

Using eq. (fT0|) . the expression for the partition function ([U) becomes 

p N 



m%). (10) 



^v,p(/3; = E /(f[ ex P (\/ |EE «p(-f M (11) 

with the standard Gaussian measure for all the z M . 

For a generic function F of the neurons, we define the Boltzmann state ojp^F) at a given level of noise (3 as the average 

u fi (F) = u(F) = (Z^GS; Or 1 E ^VK™^ (12) 

and often we will drop the subscript j3 for the sake of simplicity. Notice that the Boltzmann state does not involve the 
function fx, which factors out. The s-replicated Boltzmann state is defined as the product state f2 = lo 1 x lo 2 x ... x uj s , 
in which all the single Boltzmann states are at the same noise level and share an identical sample of quenched 
memories £. For the sake of clearness, given a function F of the neurons of the s replicas and using the symbol 
a G [1, .., s] to label replicas, such an average can be written as 

Q(F(a\ ...,a°)) = 7^ E E - E ^ -< ^ e M~P E H ^° a ' 0). (13) 

N,p a x a i (js a=1 

The average over the quenched memories will be denoted by E and for a generic function of these memories can 
be written as 

- P N n _«J^ 

nm] = / (II II \^ ) F (o = / ^(0^(0, ( 14 ) 

of course E[£f ] = and E[(£f ) 2 ] = 1. 

We use the symbol (.) to mean (.) = Efi(.). 

Recall that in the thermodynamic limit it is assumed 

lim — = a, 

a being a given real number, which acts as free parameter of the theory. 
The main quantity of interest is the intensive pressure defined as 

An*W>S) = -PfN, P (P,0 = (15) 
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while the quenched intensive pressure is defined as 

A* N , P {P) = -m, P W = -^In (16) 
and the annealed intensive pressure is defined as 

AnAP) = -PfaM = ^ IhEZ^OSjO- (17) 

According to thermodynamics, here /jv,p(/3, £) = u N,p(P,£,) — /3 _1 Sjv, p (/3, £) is the free energy density, un, p (0>Q ^ 
the internal energy density and Sjv, P (/?, £) ^ s ^ ne intensive entropy (the star and the bar denote the quenched and the 
annealed evaluations as well). Obviously, by Jensen inequality, we have A* N (0) < A^ p {(i). 



III. THE ROLE OF THE OVERLAPS AND THE INTERNAL ENERGY 

According to the bipartite nature of the Hopfield model expressed by eq. we introduce two other order 

parameters beyond the "Mattis magnetization" (eq. ([9])): the first is the overlap between the replicated neurons, 
defined as 



N 

1 



(18) 

i=l 

and the second the overlap between the replicated Gaussian variables z, defined as 

1 P 

These overlaps play a considerable role in the theory as they can express thermodynamical quantities. As an example 
let us work out the quenched internal energy of the model u% p ((3) = A^ _1 (i7jv(c; C))- 

Proposition 1 For (3 ^ 1, the quenched internal energy u* N {a,[j) of the analogical Hopfield model can be expressed 
as 

* rm _ (g*fr;0) _ (3 ( (P/N)-El(ml q i2ml) 
UN,p{P) ~ ft ~ T^~0 '' [ ' 

Proof 

The proof is based on direct calculations. We use Gaussian integration and integration by parts over the Gaussian 
memories. Let us begin with 

1 p -. v N 

N^miHxfrz)) = -2E E ("K)) + ^EE E (^ 2 ) 

fj, i 

V 

2 E /<•" ' 2 V 

Now we write explicitly a Mattis magnetization into the Boltzmann average of (f2Tj) so to use integration by parts over 
the memories £f (i.e. E£F(£) = E<%F(£)). 

--Ee(^K)) + ^ = ^EE E P(^^)) + ^ (22) 

f-i (x i 



-iE E ^K)) + 4- ( 21 ) 



R p N 1 
-2^EE E (^(^^) 2 ) " &(<nrn») + W )+^ 



^ iV 2JV 2 E( TO M9i2 m A<) D 
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In particular for (3=1 we get exactly for any N the very remarkable expression 

B m W> = |- ( 23 ) 

as it can be understood by looking at the last line of (|22[) and choosing (3 = 1. 



At the end of this section we write down a short formulary in which we consider the streaming of both the 
partition function and the Boltzmann state with respect to the level of noise (3 and to a generic stored pattern £f , as 
these calculations will be useful several times along the paper. 



. p N p 

dpz N , v m) = J2 /(II W)EE( 



'ffW (ff) 2 



x 



x exp[£E(^f^-^ 



. . 2iV 

i — 1 1 



- l^^^jm 2N~ )ZN * m) 1 ' 

i=l ii=l v r 

dqZ^m) = H\/§"(<rizn-^OZN,M0 (25) 

a fi—1 

(^exp(EE(/|^-^)))= (27) 

= l + V^E^M^) (28) 



t=i 

where, in (|25p . A € K is a generic positive real number. 

IV. THE MAIN RESULTS 

Now we are ready to state the main results of this paper, in the form of the following Theorems. 

Theorem 1 There is a (32(a) defined in the following, such that for (3 < /32(a) we have the following limits for the 
intensive free energy, internal energy and entropy, as N — > oo and p/N — ► a: 

lim (-/3f NtP (f3;$) = lim AT 1 lnZ NtP ((3; £) = In 2 - £ ln(l - /?) - ^, 
lim (uAr J ,(/3;0)=- lim N^dphLZ^iP',® 



„ M ^, I|( , vr , v/ j^o" JY '^'^ 2(1-/3)' 

hm (siv, p (/3;e)) = Mm N^QuZ^^C - (3dfskiZ N , p (/3;0) = ^ 2 - ?Ml - /?) 



j _ oV „ j , jJ , v , j ^, V __ J , JJ , V .,^ P .^__ J , rfvr .,^ 2 „ v . 2 2(1-/3)' 

^-almost surely. The same limits hold for the quenched averages, so that in particular 

lim iV^ElnZ^AO = In 2 - £ ln(l - /3) - ^. 

iV— >oo z z 
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Theorem 2 There is a (34(a) defined in the following, such that for p < Pi(a) we have the convergence in distribution 

]nZ NtP (p;£) - lnEZ N>p (/3;0 - C(j3) + X S((3) (29) 
where \ is a unit Gaussian in J\f[0, 1] and 

1 



COS) = --lnVV(l-<r 2 /3 2 a) (30) 
= (lnVlAl-^a)) 1 , (31) 

««f/l cr = (1 - 

If we consider the overlaps among s replicas, then there is a fas (<*) defined in the following, such that for (3 < fa s (a) 
the reseated overlaps converge in distribution under (.) = Ef2(.) as follows 

1/1 — 0* (3* a 

^ Fab ^ 1-^ vi ^ Xafc (33) 

where Xab and {; a b are unit Gaussian in 7V[0, 1], independent for each couple of replicas (a, b), and independent from 
the x appearing in the limit for the fluctuation of\nZ^ p . 

We remark that the limitation on the parameter regions is strictly related to our technique in the proof. There are 
good reasons to believe that the theorems can be extended to the whole expected ergodic region (3(1 + y/a) < 1. 

In order to prove these results, we need some properties about the annealed momenta of the partition function, 
that will be studied in the next section. 



V. MOMENTA OF THE NORMALIZED PARTITION FUNCTION 



Annealing is a regime in which no retrieval is achievable because it is implicitly assumed the same time-scale both for 
neurons and synapses (the synaptic plasticity is thought of as fast as the neuronal current rearrangement). Anyway, 
annealing is the first step to be investigated in order to have a good control of the statistical mechanics features of 
the model. 

To obtain the annealed intensive pressure, that we call Ajq p ((3), we must exchange the logarithm and the average 
over the memories in the expression of the quenched free energy (i.e. ElnZjv lP (/?; £) => lnEZjv,p(/3; £))■ Therefore at 
first we need to evaluate E[Zjv, p (/3; £)]. 

Proposition 2 For < (3 < 1 we have that 

EZ N , p ((3;0 = 2 N (l-(3) { -H (34) 

Proof 

EZ N , P ((3;0 



Then, by recalling the definition ([5]) of Z^ tP , we can immediately state the following 

Proposition 3 In the thermodynamic limit, and for every value < (3 < I, the annealed pressure per site A(a,(3) = 
lirn/v^oo An,p((3) of the analogical Hopfield model is 

lim ±lnE(Z N , p (ftO)=A(a,p)=\n2-^ln(l-(3)-^. ( 35 ) 



r- V N - p p N rr 

/(nn^)) / n^)E ex p(EE(VA7^ zM)) 

E All ^))exp((/3/2) E4) = ^ Z 3 )" 1 ■ 

_ •/ 1 1 
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Remark 1 We stress that the annealed free energy H35\) turns out to be the same as in the AGS theory with digital 
patterns. Furthermore, 

lim A(a,j3) = ln2 + {fi'f/A, (36) 

Q-»oo,/3^0,\/a/3-»/3' 

that is the expression for the annealed free energy of the Sherrington- Kirkpatrick model at the temperature f3' ]3_ 

This is not surprising as, at given N, in the limit p — > oo, (3 — ► with /3^/p/N — ► (3' we get in distribution 
Zjv> (/?;£) — > Z^ K (f3'; J), where Sif stands for the Sherrington-Kirkpatrick model and J is the associated noise 
(standard unit Gaussian Jy for each couple of sites). At given N, the neural network with infinite stored patterns 
becomes a Sherrington-Kirkpatrick mean field spin glass. 

Remark 2 If we now turn to the energy density it is easy to show that its "annealed expression" , defined as u(a, (3) = 
—dpA(a, 0), is given by 

lim u NtP (0) = - lim \nEZ Np = ~ a , (37) 

where, with respect to \20\) thought in the infinite volume limit, the order parameters are missing. 

Both the expressions (|35I37[) do diverge in the limit of (3 — ► 1 suggesting a point where annealing has to break, 
whatever a. 

To investigate the region of validity of the annealing, which is defined as the ergodic region, we have to study the 
momenta of the partition functions and check if and where they are well defined. 
It will be easier to deal with the normalized partition function Zjv lP (/3; £) defined as 

Z Np m)= Z ?* m) . (38) 

As the momenta will turn out to be expressed in terms of overlaps, the following Lemma will be of precious help for 
our purpose. 

Lemma 1 At (3 = 0, in the thermodynamic limit the two replica overlaps q a f,, a ^ b, become almost surely zero and 
their rescaled values yNq a b converge in distribution to unit Gaussian £ a &, independent for each couple of replicas. 

Proof 

At (3 = the Boltzmann measure fio becomes flat on all the configurations, so that all afs are independent and take 
the values ±1 with equal probability 1/2. By taking into account the expression of the overlap q a b = N^ 1 of of , 
the Lemma follows by the strong law of large numbers and the central limit theorem on sums of independent variables. 
□ 

Theorem 3 For s £ N and for (3 < (3 s {a), with (3 s {a) suitably defined in the following, the following limiting 
expression for the integer momenta of the normalized partition function holds 

lim nZh, p m)] =exp(fc^(ln(- ^-))), (39) 

where a = 1/(1 — f3). 
Proof 

Let us at first notice that there is a constraint f3 s {a) < s^ 1 , which becomes effective starting from s = 2. The key 
point is that E[Zjy (/?;£)] becomes infinite, at fixed N, if the constraint is not satisfied. In fact, let us calculate this 
momentum, by introducing the replicated erf, z", for i = 1, 2, N, /J, = 1, 2, ...,p, a = 1,2, s. 

p N 

EZ s N Jf3;0 Z^" 



. s p p N r~r s 

(£■■■£ An II d ^D) «p { £ £ (v & £ }) 

cr 1 cr s a—1 fj,= 1 fi—1 i—1 a—1 

p s p R p N s 

- £•••£ Ann^))-p(4£B£^) 2 ) 

cr 1 (7 s a—1 — 1 /.i—1 i—1 a—1 

r P s a N s 

= £-£ An)((n *w)«p(^E(E^ 

cr 1 cr s /i—1 a—1 i—1 a, 6 



(40) 
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where we have performed the Gaussian integration over the memories £, and have expressed the resulting square in 
the form of the double sum ^2 a b - The sum over i gives the overlaps q a \,. We notice also that we have complete 
factorization over /i. Therefore, we can introduce generic variables z a , a = 1, 2, s, and define 

B P ,M Q) = /(II M* a )) exp(^ J2 V"bz a z b ), (41) 

J a=l a,b 

where Q is the s x s matrix with elements q a b, so that 

EZ s Np (f3;Z) = 2 Ns n (e phlB "-^ Q) ). (42) 
Now we can see the reason for the limitation (3 < 1/s. In fact, let us notice that 

< 5> fc z^ < £ \z*\\z»\ = S *( S -^\z a \) 2 < s 2 s-^ a f = -E^) 2 ' 

a,b a,b a a a 

where we have introduced the uniform distribution s~ 1 ^ Q on (1, 2, s), and exploited Schwartz inequality. There- 
fore, the integral defining B PtS (/3;Q) in ([41]) can be uniformly bound by (1 — sf3)~ p / 2 , which is finite in the region 
s/3 < 1. On the other hand, it is easily seen that for some a configurations the integral in (|4Tj) is infinite if s/3 > 1. 
The worst case is when all q a b = 1- Then we have from (|4ip 

B P ,s = /(n^ a ))exp(|E^)= /(n^ a ))ex P (^G>> a ) 2 ) 

a— 1 a.b a— 1 a 

= f (f[df,(z a )) / rf/i(y)exp(^E zQ f) = / ^(y)exp(^y 2 ), 

a— 1 ' a 

and the integral on the auxiliary variable y is divergent if s/3 > 1. From now on we remain in the region s[3 < 1. 
Let us go back to the definition (|4"Tj) . Write ^ Q b q a bZ a z b = 2J2(ab) QabZ a z b + J2 a (z a ) 2 , where (ab) are all couples of 
different replicas. Then we have 

B P ,MQ) = (l[(dfi(z a )e 1 ^ 2 ))eMPT,^z a z b ) 

o=l (ab) 

= ( / d/i(Oe^') 2 ) s / (n^(z a ))exp(/?E9a^^ b ) 

^ a=l (ab) 

= (l-/?r 4 /(n^(^))exp(/?E^ a ^), (43) 

^ Q=l (ab) 

where we have introduced the normalized deformed measure 

_ e^ z2 dn(z) 

Finally, if we define a modified B p ^ s (f3; Q) as 

p s s 

B P , S (P;Q)= (l[dfl(z a ))exp(P^2 ( labZ a z b ), (44) 

J a=l (ab) 

we can write 

EZ' NtP (/3;Z) = (Ei W)P (/3;0) s Oo(e pln ^^^«). (45) 

In order to investigate the N — > oo limit, it is convenient to start from the case where s = 2. Then B P 2 can be 
explicitly calculated in the form 

S Pl2 = ~]n(l-/3VW (46) 
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In fact, in this case we have 

B p ,2= / dp,(z 1 )dfi(z 2 ) exp({3qi 2 z 1 z 2 ), 



where the two Gaussian integrals can be calculated explicitly and lead to (|46j) . Therefore, we are led to consider the 
(a, 0) region where 

Qo(exp(-ipln(l-/3 2 a 2 gi 2 2 ))) 

stays finite in the N — > oo limit. Through a simple change of variables o~\ = c^a 2 = cr[cri we are led to the 
consideration of a mean field ferromagnetic Ising system (o"i, er/v) with normalized partition function 

tt (exp(-ipln(l-/?Vm 2 ))), (47) 
where now flo = 2~ N ^2 a , and m = N^ 1 <7j, as usual. Now, we can state and prove the following Theorem. 
Theorem 4 Consider the trial function 

0(a, ft M) = -la ln(l - /?VM 2 ) + In co S h(« x ^ M 2 ) - ^ 77^ ^ 

depending on the order parameter M, with —1 < M < 1. Clearly, at M — we /iave (/>(a,/3;0) = 0. Define /^(ct) as 
the largest value such that, for any (3 < ^(a), we have <fi(a,/3;M) < for any positive M. It is easily shown that 
$2(0:) > (1 + VI + a) -1 . ITien /or /? < /92(a) we /iai>e 

lim Q (exp(— jjlnfl - (3 2 a 2 m 2 ))) = (1 - a/jV)"'. (48) 

Notice that /?2 (a) defines the onset of the ferromagnetic phase transition for the model. The proof of the Theorem 
follows standard methods of statistical mechanics. In order to get a lower bound we only notice that 

-ln(l- (3 2 a 2 m 2 ) >[3 2 a 2 m 2 , 

and therefore 

n (exp(-ipln(l - /?Vm 2 ))) > n (exp(ip/?Vm 2 )). 

The r.h.s. converges to (1 — a{3 2 a 2 )~? , provided a/? 2 cr 2 < 1. Notice that this last condition can be written also as 
(3 < 1/(1 + \/a), which is the critical line according to the AGS theory. For the upper bound, let us introduce the 
truth functions on the a configuration space, defined by \i = x( m2 < "i 2 ); and \i — x( m2 > "^ 2 )) where fh is some 
positive number. Then we have that the Oq(...) in (|4T|) splits into the sum of two pieces 

ti (...) = n (... X i) + n (... X 2)- 

For (3 < /32(a), the second piece converges to zero as N — ► 00. In fact, let us define for — 1 < M < 1 

^(M) = -^ln(l-/?VM 2 ), (49) 

with its M derivative 

Notice that -0 is convex in M, so that 

^(m) > ip(M) + (m - M)^'(M). (51) 
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Now, let us consider M as taking all values of m for which m 2 > m 2 . There are at most N + 1 such values. For the 
sake of simplicity, introduce the inequality 



1 < 

M 



^ exp ( - N(ip(m) - ip(M) - (m - M)ip'(M))) , (52) 



where M is summed over all stated values. The reason of the inequality is clear. In fact, there is one term equal to 
1, when M = m, while all other terms are positive. By exploiting the inequality, we can write 

fio(exp(-ipln(l - /3Vm 2 )) X2 ) < £ fi (exp ( - N( - ip(M) - (m - M)V'(M))) X2 ). (53) 



If now we remove the xi-, an d perform the average over Slo, by taking into account that the exponent is factorized 
with respect to the oVs, we get 

no(exp(-ij»ln(l - (3 2 a 2 m 2 )) X2 ) < £ exp (n(^(M) + In coBh(|= 1 f Zjf M2 ) - M^'(M))). (54) 



Clearly, in the region /3 < 02(a) and for large N, each term in the sum is uniformly bounded by an exponential of 
the type exp(— cN), for some constant c. Of course there are at most N + 1 terms. Therefore, as N — > oo, the r.h.s. 
goes to zero, as we were interested to show. 

Now we must consider the first term f2o(---Xi)- Let us notice that in the region m 2 < fh 2 by convexity we have 
— ln(l — f3 2 a 2 m 2 ) < — ln(l — f3 2 a 2 fh 2 )(m 2 /fh 2 ). By inserting the inequality in the first term, and neglecting the \i> 
we have in the infinite volume limit 

hmsu P r!o(exp(-^ln(l - /?Vm 2 )) Xl ) < (1 + - ^ ~ yi, (55) 

Since m is arbitrary, we can take fh — > 0. Collecting all results, we immediately establish the limit in (I48p . Notice 
that /?2(a) < (1 + yfa)' 1 , otherwise <j>(a,(3;M) would start with a positive derivative at M — 0. This ends the proof 
of Theorem [3] in the case s = 2. 

The general case can be handled in a similar way. Now we encounter ferromagnetic models for the Ising variables 
<jf,a — l,2,...,s,i = 1,2, ...,7V with Boltzmannfaktor exp(p\nB ps ). If /3 S ( Q ) denotes the onset of the associated 
ferromagnetic transition, then we can immediately prove that 

lim O ( e ^,s)= ex p(fc^(m(- ^55-))), (56) 

for (3 < /3 S ( Q ). In fact, as in the proof for the case s = 2, we see that in the expression of B PjS (f3; Q) only the first two 
terms in the expansion of the exponent do matter, provided the stated condition on [3 holds. These terms are easily 
calculated as in the case s = 2. Then we recall that under Oo, for a ^ b, the rescaled overlaps ^/~Nq a b converge in 
distribution to independent unit Gaussian £ a {,. We see that the term s(s — 1) in formula (|39p comes essentially from 
the fact that there are s(s — l)/2 couples (a, b) for s replicas. Therefore, Theorem|3]is fully established. □ 

Now we are ready to prove Theorem [TJ at least in the region j3 < faia). 
First of all, let us recall that if u^r > is a sequence of random variables normalized to E(itjv) = 1, then a simple 
application of the Markov inequality [3(| and the Borel-Cantelli Lemma gives 

lim sup — lnitjv < 0, 

almost surely. Moreover, if E(u 2 v ) < cE 2 (un), uniformly in N , for some finite constant c, then 

lim -i lnujv = 0, 

JV^oo N 

almost surely. 
If we define 

E(Z N , p (f3;t) s 

u N = 
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and take into account our previous results, then we immediately find, ^-almost surely 

lim sup i- In Z Njt (J3', < In 2 - ^ m (i _ for any /? < 1, (57) 

1 a 

lim — lnZjv iP (^;0 = In 2 - - ln(l - /?), for any < /3 2 (a). (58) 

N—>oc IV z 

In order to get Theorem [TJ in the stated region, it is only necessary to recall the equation |(7J) connecting Z^.p with 
Zjv, p , and the limiting properties of /jy. □ 



VI. LOG-NORMALITY OF THE LIMITING DISTRIBUTION FOR THE PARTITION FUNCTION 

In this section we want to show that the limiting distribution of the normalized partition function (|38[) . at least in 
a given region of the a, (3 plane, is log-normal. This will immediately give us the mean and the fluctuations of the 
thermodynamical quantities in that region (29j . 

Let us remember that if C(f3) and S(0) are given functions and x a standard gaussian 7V[0, 1], a variable r)(/3) has a 
log-normal distribution if it is possible to write it down as [l6| 

V (j3)=eKp{C(0)+xS(P)). (59) 



The momenta of w{(3) are 



E[n s (0)} = E[cxp( s C(/3) + sS(f3) X )} = exp(C(/3) s + \s 2 S 2 {0)). (60) 



So if we choose 



s ' m = Wi^W (62) 

we see that Zjv, p (/3;£) and ?y(/3) have the same integer momenta in the limit, provided we restrict the values of /3, 
according to the order of the momentum s, as expressed in Theorem [3l This seems to suggest that Zm, p (0',C) is 
log-normal distributed in the limit. To prove that this is effectively the case it will be sufficient to prove that the 
momenta Zn, p {0] £) a do in fact converge to those of r\ for all values of A in some interval of the real line [T3|. In 
other words, we have to extend the limiting behavior of Theorem [3J from integer values of s to real values A in some 
nontrivial interval, at least in some region of the (a, (3) plane. To solve this task we have to evaluate the limiting 
behavior of p {(3; £), for A in some interval of the real line. 
We get the result by analyzing 

d f3 lnE[Z*J = tyOuE^J - AlnEfej) = - A^ffi^l, (63) 

H z n, p \ H z n, p \ 

that can be written in terms of overlaps via the following helpful Proposition. 

Proposition 4 For any real X, with A < s, s integer, and [3 < l/s, the (3-derivative of the annealed real momenta of 
the partition function can be expressed in terms of overlaps as 

Notice that 9/3lnE[Z^ p ] is convex increasing in A. Therefore, the limitation on the values of assures the existence 

of the relevant averages. 

Proof 

Using equation (|24p we can write 

p N . 

d fJ nz x N j = nxz^dpz^] -EE (^f E Kf4/(^/)i) • (65) 
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Furthermore we can write 



EftfZ^faz")] = \^nZ^y(a^)]~X^n^Z x N ^{a^)] (66) 

N 



3 = 1 

and, using (|25l26l27p . 

p N 



^1 E[Z^ p ] + IJ2 E ^nJH^)] ^EiZ^ia^)}, 



£ £ E^Z^a^)} = {l _ P /^ x/N) (( A - lWZ^ p n(q 12Pl2 )} + E[ZkJ) . (67) 



fi— 1 i— 1 

7.\ 



By substituting (|67|) into (|65|) and dividing by E(-Zjv p ) we get the result. □ 



With the help of ([51)) we can rewrite (|63[) as 

3 3 lnE[^g = ^zi) E ( 5 ^n(ViV gl2 ^ 12 )). (68) 

To proceed further we have now to investigate the distribution of the rescaled overlaps because they do appear into 
the expression above. Such distribution can be obtained trough the evaluation of their momenta generating function. 
We will see that at least in a given region the distribution of Z^ p is not coupled with the one of the overlaps. From 
this observation, by looking at eq. (|69[) the log-normality for the normalized partition function is easily achieved. 
Let us start by proving the following 

Proposition 5 Consider a generic number of replicas s. For each couple (a, b) of replicas, let (X ab ,rj ab ) be real 
numbers in the momentum generating functional (which we assume to be very small). Let X be a real number in the 
interval s < X < 2s. Then, at least for (3 < /3 2s (a;), we have the limit 



^E(^—^n(exp{^X ab VNq ab + ^ Vab ^ab))) (69) 

I N,P' ab ab 



= exp(-^A afc i _ +-^ Vab a ( 1 „ (t2/?2q +1) + L \ 

ab ab ab 

where as usual a = 1/(1-/3) and denotes the sum over all couples (ab). 
Proof 

We give the proof at first for A = s, in the region (3 < f3 s (a). Then we will enlarge the proof to the interval s < X < 2s, 
in the region (3 < /3 2s (a). For X = s the l.h.s. of (|6"9"|) can be thought of as 

(TIE) |(n rfM ^ ))e ( fE - iE -^^0e( E - A " 6( ^ )+2: -'' a6 ^) = 

ia erf Lta 

(II E) /(II^«)) ex p(E 

y/pPabiVaP^ab + Vab)j 6Xp(^ Xab^a^Cr" 2 = 
ia af fia ab ab 

tt (exp (£, l<r 2 (aP 2 £b + Vab + ^PVabU))) exp(^ X ab U)^2 Ns = 

ab ab 

CX P (I E l-l^R2 a ^ 2(jirt -b + X 'b + ^(3a 2 X ab i lab ) + ^rf ab )nZ s NtP l 

ab 

where d\i a is the Gaussian with variance u = (1 — j3)~ . Therefore, by taking the limit, we prove the proposition 
X = s. In order to provide the extension to the interval s < X < 2s we must show that defining 

- ex P (- £ X ab — — + - £ rfo {^—^ + 1) + E Y^oWa K ^ 

ab ab ab 
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the following holds 

lim e(z* 0(exp(V X ab VNq ab + V VabVPPab) - A)) = 0. (70) 



JV- _ . 

ab ab 



The proof of (|70|) can be obtained in the simplest way by using Cauchy-Schwartz inequality (E 2 [Ai?] < EL4 2 ]E[£> 2 ]), 
choosing A = fj, + s, with < /i < s. In fact, we have 

E 2 lz^ p Z^ p ln(exp(J2KbVNq ab + J2vabVWab))-A) j < 

V V ab ab II 

E (ZftjE \Z^ p ( fi(cxp(£ KbVNq ab + £ VabVPPab)) ~ a] ) . (71) 



a6 



Here the first factor is bounded in the region (3 < /32s (a). In fact, by monotonicity we have E{Z 2 l ^' p ) < E(Z^ p ). The 
second term is the sum of three terms obtained by calculating the square of 

(^(exp(E afc KbVNqab + ^abVab^/pPab)) - Aj , the simplest being E(Z^ p )A 2 which is known. 
It is easy to check that for the double-product we have in the limit 



lim AE Z#_ft(exp(> \a b VNq ab +\riaby/pPab))] = Hm A*E(Zf? ). 

\ ab ab / 

For the last term we have 

E ZN^(eMYs Xab ^Q ab + ^VabVPPa^ 
\ ab ab 

lim = A 2 E{Z 2 N \ p ), (72) 

where the state O is thought of by 2s replicas, the sum on the couples of variables (a, b) taking into account the second 
set s + 1, ...,2s. 

The sum of these three terms goes to zero as N — > oo proving the Proposition. □ 

From Proposition ([5]) we can derive the next Corollary, which is a part of Theorem [5] 

Corollary 1 For s replicas, at least for p < fosict), in the thermodynamic limit the limiting distribution of the 
rescaled overlaps are 

\'<l,i. U 1 7'! ) 



VpPab -> " J^tab + q ^Xab (74) 



where \ab e W(0, 1) and £ ab G 7V(0, 1/(1 - a 2 p 2 a)). 

Now we are ready for the proof of the following basic Theorem. 

Theorem 5 For (3 < /34(a), in the thermodynamic limit, we have that in distribution 

Zn.M exp (C{[3) + X S((3)) (75) 

where x S A/"[0, 1] and 

COS) - -ilnVl/(l-^ 2 /3 2 a) (76) 



S(/3) = lnVVa-^a) . (77) 
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Proof 

The limitation j3 < (3i(a) is clear. In fact, we will exploit formula which requires two replicas, for 2 < A < 4, 
and the results of Proposition ([5]), that require the stated limitation on (3. Therefore, we see immediately that 



lim E(z*n{VNq ab VPPab)/HZN P )) (78) 

I— >oo \ ^ / 

.abVNqal 



E ( Z N p fi ( eX P(Eab ^ab\fN q ab + J^ab Vaby/PPab)) , 

= d Xab d nab lim — ^ L (79) 



\faj3a 



(1 -cr 2 /? 2 a)' 



Therefore, we have 



By exploiting convexity in /3, we can integrate this expression and obtain the limit 

lim E[Z^]^exp(^-^(ln(- \— 

N— too ' y \ 4 1 — a*p A a 



(80) 



for all values of A in a nontrivial interval of the real axis. This shows the convergence in distribution of Z^, p to a 

log-normal random variable, as stated in the Theorem. □ 

Finally, we can easily prove Theorem [2] if we recall the definition in ([55)1 . □ 



VII. CONCLUSION 



In this work we introduced the framework of the real replicas, successfully applied on spin-glasses (see e.g. 
PI [HI [23]), to neural networks in the ergodic regime. This approach naturally holds for the high storage memory case, 
which is mathematically challenging. Acting together as a biological generalization to analogical stored memories and 
as a technical trick to manage easily the mathematical control, we allowed the patterns to live as Gaussian variables 
on 7V[0, 1] instead of ±1 but, as we checked a fortiori, this does not affect (at least in the part of the ergodic region 
that we can control) any macroscopical distribution once the thermodynamic limit is taken. Thinking at the Hopfield 
model as a bipartite model in a proper space of variables, beyond the Mattis magnetization, we introduced the other 
order parameters q a b and p a b, one for each interacting structure, the N dichotomic Ising neurons Oi and the p fictitious 
Gaussians z M , which are able to fully describe the high temperature region we investigated. 

We showed that the partition function is log-normal distributed in a suitably defined regionn, then we evaluated the 
distribution of the rescaled overlaps, which share centered Gaussian fluctuations with different variances. Finally we 
proved that all the thermodynamic quantities fluctuate around their annealed approximation and calculated their 
spread. All the densities (e.g. energy density, free energy density and entropy density) turn out to be self-averaging 
on their annealed values. 

Further investigation should give us the full control of the whole ergodic phase and bring us exploring the retrieval 
phase and hopefully the still completely obscure broken replica phase. 
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